Risk, Unexpected Uncertainty, and Estimation Uncertainty: Bayesian Learning in Unstable Settings

Recently, evidence has emerged that humans approach learning using Bayesian updating rather than (model-free) reinforcement algorithms in a six-arm restless bandit problem. Here, we investigate what this implies for human appreciation of uncertainty. In our task, a Bayesian learner distinguishes three equally salient levels of uncertainty. First, the Bayesian perceives irreducible uncertainty or risk: even knowing the payoff probabilities of a given arm, the outcome remains uncertain. Second, there is (parameter) estimation uncertainty or ambiguity: payoff probabilities are unknown and need to be estimated. Third, the outcome probabilities of the arms change: the sudden jumps are referred to as unexpected uncertainty. We document how the three levels of uncertainty evolved during the course of our experiment and how it affected the learning rate. We then zoom in on estimation uncertainty, which has been suggested to be a driving force in exploration, in spite of evidence of widespread aversion to ambiguity. Our data corroborate the latter. We discuss neural evidence that foreshadowed the ability of humans to distinguish between the three levels of uncertainty. Finally, we investigate the boundaries of human capacity to implement Bayesian learning. We repeat the experiment with different instructions, reflecting varying levels of structural uncertainty. Under this fourth notion of uncertainty, choices were no better explained by Bayesian updating than by (model-free) reinforcement learning. Exit questionnaires revealed that participants remained unaware of the presence of unexpected uncertainty and failed to acquire the right model with which to implement Bayesian updating

The Affective Impact of Financial Skewness on Neural Activity and Choice

Few finance theories consider the influence of “skewness” (or large and asymmetric but unlikely outcomes) on financial choice. We investigated the impact of skewed gambles on subjects' neural activity, self-reported affective responses, and subsequent preferences using functional magnetic resonance imaging (FMRI). Neurally, skewed gambles elicited more anterior insula activation than symmetric gambles equated for expected value and variance, and positively skewed gambles also specifically elicited more nucleus accumbens (NAcc) activation than negatively skewed gambles. Affectively, positively skewed gambles elicited more positive arousal and negatively skewed gambles elicited more negative arousal than symmetric gambles equated for expected value and variance. Subjects also preferred positively skewed gambles more, but negatively skewed gambles less than symmetric gambles of equal expected value. Individual differences in both NAcc activity and positive arousal predicted preferences for positively skewed gambles. These findings support an anticipatory affect account in which statistical properties of gambles—including skewness—can influence neural activity, affective responses, and ultimately, choice

Interoceptive inference, emotion, and the embodied self

The concept of the brain as a prediction machine has enjoyed a resurgence in the context of the Bayesian brain and predictive coding approaches within cognitive science. To date, this perspective has been applied primarily to exteroceptive perception (e.g., vision, audition), and action. Here, I describe a predictive, inferential perspective on interoception: ‘interoceptive inference’ conceives of subjective feeling states (emotions) as arising from actively-inferred generative (predictive) models of the causes of interoceptive afferents. The model generalizes ‘appraisal’ theories that view emotions as emerging from cognitive evaluations of physiological changes, and it sheds new light on the neurocognitive mechanisms that underlie the experience of body ownership and conscious selfhood in health and in neuropsychiatric illness

Under pressure: Response urgency modulates striatal and insula activity during decision-making under risk

When deciding whether to bet in situations that involve potential monetary loss or gain (mixed gambles), a subjective sense of pressure can influence the evaluation of the expected utility associated with each choice option. Here, we explored how gambling decisions, their psychophysiological and neural counterparts are modulated by an induced sense of urgency to respond. Urgency influenced decision times and evoked heart rate responses, interacting with the expected value of each gamble. Using functional MRI, we observed that this interaction was associated with changes in the activity of the striatum, a critical region for both reward and choice selection, and within the insula, a region implicated as the substrate of affective feelings arising from interoceptive signals which influence motivational behavior. Our findings bridge current psychophysiological and neurobiological models of value representation and action-programming, identifying the striatum and insular cortex as the key substrates of decision-making under risk and urgency

Rapid Processing of Both Reward Probability and Reward Uncertainty in the Human Anterior Cingulate Cortex

Reward probability and uncertainty are two fundamental parameters of decision making. Whereas reward probability indicates the prospect of winning, reward uncertainty, measured as the variance of probability, indicates the degree of risk. Several lines of evidence have suggested that the anterior cingulate cortex (ACC) plays an important role in reward processing. What is lacking is a quantitative analysis of the encoding of reward probability and uncertainty in the human ACC. In this study, we addressed this issue by analyzing the feedback-related negativity (FRN), an event-related potential (ERP) component that reflects the ACC activity, in a simple gambling task in which reward probability and uncertainty were parametrically manipulated through predicting cues. Results showed that at the outcome evaluation phase, while both win and loss-related FRN amplitudes increased as the probability of win or loss decreased, only the win-related FRN was modulated by reward uncertainty. This study demonstrates the rapid encoding of reward probability and uncertainty in the human ACC and offers new insights into the functions of the ACC

From uncertainty to reward: BOLD characteristics differentiate signaling pathways

<p>Abstract</p> <p>Background</p> <p>Reward value and uncertainty are represented by dopamine neurons in monkeys by distinct phasic and tonic firing rates. Knowledge about the underlying differential dopaminergic pathways is crucial for a better understanding of dopamine-related processes. Using functional magnetic resonance blood-oxygen level dependent (BOLD) imaging we analyzed brain activation in 15 healthy, male subjects performing a gambling task, upon expectation of potential monetary rewards at different reward values and levels of uncertainty.</p> <p>Results</p> <p>Consistent with previous studies, ventral striatal activation was related to both reward magnitudes and values. Activation in medial and lateral orbitofrontal brain areas was best predicted by reward uncertainty. Moreover, late BOLD responses relative to trial onset were due to expectation of different reward values and likely to represent phasic dopaminergic signaling. Early BOLD responses were due to different levels of reward uncertainty and likely to represent tonic dopaminergic signals.</p> <p>Conclusions</p> <p>We conclude that differential dopaminergic signaling as revealed in animal studies is not only represented locally by involvement of distinct brain regions but also by distinct BOLD signal characteristics.</p

Gain and Loss Learning Differentially Contribute to Life Financial Outcomes

Emerging findings imply that distinct neurobehavioral systems process gains and losses. This study investigated whether individual differences in gain learning and loss learning might contribute to different life financial outcomes (i.e., assets versus debt). In a community sample of healthy adults (n = 75), rapid learners had smaller debt-to-asset ratios overall. More specific analyses, however, revealed that those who learned rapidly about gains had more assets, while those who learned rapidly about losses had less debt. These distinct associations remained strong even after controlling for potential cognitive (e.g., intelligence, memory, and risk preferences) and socioeconomic (e.g., age, sex, ethnicity, income, education) confounds. Self-reported measures of assets and debt were additionally validated with credit report data in a subset of subjects. These findings support the notion that different gain and loss learning systems may exert a cumulative influence on distinct life financial outcomes

An effect of serotonergic stimulation on learning rates for rewards apparent after long intertrial intervals

Serotonin has widespread, but computationally obscure, modulatory effects on learning and cognition. Here, we studied the impact of optogenetic stimulation of dorsal raphe serotonin neurons in mice performing a non-stationary, reward-driven decision-making task. Animals showed two distinct choice strategies. Choices after short inter-trial-intervals (ITIs) depended only on the last trial outcome and followed a win-stay-lose-switch pattern. In contrast, choices after long ITIs reflected outcome history over multiple trials, as described by reinforcement learning models. We found that optogenetic stimulation during a trial significantly boosted the rate of learning that occurred due to the outcome of that trial, but these effects were only exhibited on choices after long ITIs. This suggests that serotonin neurons modulate reinforcement learning rates, and that this influence is masked by alternate, unaffected, decision mechanisms. These results provide insight into the role of serotonin in treating psychiatric disorders, particularly its modulation of neural plasticity and learning.info:eu-repo/semantics/publishedVersio

Bayesian Integration and Non-Linear Feedback Control in a Full-Body Motor Task

A large number of experiments have asked to what degree human reaching movements can be understood as being close to optimal in a statistical sense. However, little is known about whether these principles are relevant for other classes of movements. Here we analyzed movement in a task that is similar to surfing or snowboarding. Human subjects stand on a force plate that measures their center of pressure. This center of pressure affects the acceleration of a cursor that is displayed in a noisy fashion (as a cloud of dots) on a projection screen while the subject is incentivized to keep the cursor close to a fixed position. We find that salient aspects of observed behavior are well-described by optimal control models where a Bayesian estimation model (Kalman filter) is combined with an optimal controller (either a Linear-Quadratic-Regulator or Bang-bang controller). We find evidence that subjects integrate information over time taking into account uncertainty. However, behavior in this continuous steering task appears to be a highly non-linear function of the visual feedback. While the nervous system appears to implement Bayes-like mechanisms for a full-body, dynamic task, it may additionally take into account the specific costs and constraints of the task

Spatiotemporal neural characterization of prediction error valence and surprise during reward learning in humans

Reward learning depends on accurate reward associations with potential choices. These associations can be attained with reinforcement learning mechanisms using a reward prediction error (RPE) signal (the difference between actual and expected rewards) for updating future reward expectations. Despite an extensive body of literature on the influence of RPE on learning, little has been done to investigate the potentially separate contributions of RPE valence (positive or negative) and surprise (absolute degree of deviation from expectations). Here, we coupled single-trial electroencephalography with simultaneously acquired fMRI, during a probabilistic reversal-learning task, to offer evidence of temporally overlapping but largely distinct spatial representations of RPE valence and surprise. Electrophysiological variability in RPE valence correlated with activity in regions of the human reward network promoting approach or avoidance learning. Electrophysiological variability in RPE surprise correlated primarily with activity in regions of the human attentional network controlling the speed of learning. Crucially, despite the largely separate spatial extend of these representations our EEG-informed fMRI approach uniquely revealed a linear superposition of the two RPE components in a smaller network encompassing visuo mnemonic and reward areas. Activity in this network was further predictive of stimulus value updating indicating a comparable contribution of both signals to reward learning